Abstract MDP Reward Shaping for Multi-Agent Reinforcement Learning

MDP Reward Shaping for Multi-Agent Reinforcement Learning Kyriakos Efthymiadis, Sam Devlin and Daniel Kudenko Department of Computer Science, The University of York, UK Abstract. Reward shaping has been shown to significantly improve an agent’s performance in reinforcement learning. As attention is shifting from tabula-rasa approaches to methods where some heuristic domain knowledge can be given to agents, an important problem that arises is how to generate a useful potential function. Previous research demonstrated the use of plan-based reward shaping in multi-agent reinforcement learning (MARL), where STRIPS planning was used to generate a potential function. The results showed that potential functions based on joint plans can improve an agent’s performance. When using individual plans however, the agents face conflicting goals which can have a detrimental effect in performance. In this paper we present the use of abstract MDPs as a method to provide heuristic knowledge in MARL and how it can be utilised for conflict resolution when agent communication and goal-shaping is not possible. Reward shaping has been shown to significantly improve an agent’s performance in reinforcement learning. As attention is shifting from tabula-rasa approaches to methods where some heuristic domain knowledge can be given to agents, an important problem that arises is how to generate a useful potential function. Previous research demonstrated the use of plan-based reward shaping in multi-agent reinforcement learning (MARL), where STRIPS planning was used to generate a potential function. The results showed that potential functions based on joint plans can improve an agent’s performance. When using individual plans however, the agents face conflicting goals which can have a detrimental effect in performance. In this paper we present the use of abstract MDPs as a method to provide heuristic knowledge in MARL and how it can be utilised for conflict resolution when agent communication and goal-shaping is not possible.